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Abstract 

Background: Systematic reviews have the potential to inform decisions made by health policymakers and 
managers, yet little is known about the impact of interventions to increase the use of systematic reviews by these 
groups in decision making. 

Methods: We systematically reviewed the evidence on the impact of interventions for seeking, appraising, and 
applying evidence from systematic reviews in decision making by health policymakers or managers. Medline, 
EMBASE, CINAHL, Cochrane Central Register of Controlled Trials, Cochrane Methodology Register, Health 
Technology Assessment Database, and LISA were searched from the earliest date available until April 2010. Two 
independent reviewers selected studies for inclusion if the intervention intended to increase seeking, appraising, or 
applying evidence from systematic reviews by a health policymaker or manager. Minimum inclusion criteria were a 
description of the study population and availability of extractable data. 

Results: 1 1,297 titles and abstracts were reviewed, leading to retrieval of 37 full-text articles for assessment; four of 
these articles met all inclusion criteria. Three articles described one study where five systematic reviews were 
mailed to public health officials and followed up with surveys at three months and two years. The articles reported 
from 23% to 63% of respondents declaring they had used systematic reviews in policymaking decisions. One 
randomised trial indicated that tailored messages combined with access to a registry of systematic reviews had a 
significant effect on policies made in the area of healthy body weight promotion in health departments. 

Conclusions: The limited empirical data renders the strength of evidence weak for the effectiveness and the types 
of interventions that encourage health policymakers and managers to use systematic reviews in decision making. 



Background 

Policymakers and managers working within health sys- 
tems make decisions in efforts to improve health for 
individuals. The impact of the choices made by policy- 
makers is experienced in the health status and daily 
lives of people in the form of laws and regulations, 
guidelines, public education campaigns, among others 
[1]. The choices made by healthcare managers affect 
environments where common goals and strategies must 
be found between clinical and administrative 
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environments [2]. Overall, decisions by policymakers 
and managers are made around burdensome health pro- 
blems, within complex health systems, and ideally 
involve effective solutions and strategies to support their 
implementation. 

Increasingly, systematic reviews are seen as helpful 
knowledge support for policymakers and managers 
[3-6]. Systematic reviews of effects are concise summa- 
ries that address sharply defined questions, employing 
rigorous methods to select credible and relevant infor- 
mation in order to generate summative reports [4,7]. 
The review was carried out in two stages: 1) a formal 
scoping review (a method for mapping existing literature 
in a topic area and identifying gaps [8]) to understand 
the extent to which evidence from systematic reviews is 
sought, appraised, understood, and used to inform 
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decision-making in four key areas: clinical practice, 
health systems management, public health, and policy 
making; and 2) a systematic review to determine the 
impact (a change identified by individual perception or 
by quantification) on professional performance and 
healthcare outcomes of interventions for seeking, 
appraising, and applying evidence from systematic 
reviews in decision making by health policymakers and 
managers that is reported in this manuscript. 

Methods 

Data sources and searches 

The databases of Medline (1950 to April 2010), 
EMBASE (1980 to April 2010), CINAHL (1982 to April 
2010), Cochrane Central Register of Controlled Trials 
(CENTRAL) (to April 2010), Cochrane Methodology 
Register (to April 2010), Health Technology Assessment 
Database (to April 2010), and LISA (Library and Infor- 
mation Science Abstracts) (1969 to April 2010) were 
searched using the terms systematic review, meta analy- 
sis, evidence synthesis, methodologic review, and quanti- 
tative review combined with implement, use, utilize, 
seek, retrieve, appraise, and apply (see Additional File 1). 
The grey literature was searched after identifying key 
websites and search engines, such as Google and Intute. 
Reference lists of all papers and relevant reviews were 
screened for any further published or unpublished work 
and experts in the field were contacted to identify any 
further studies. No language restrictions were placed on 
the search strategy. 

Study Selection 

We included all study designs except qualitative studies. 
For this study, a health policymaker was defined as an 
individual elected or appointed to office at some level of 
government. A health manager was defined as an indivi- 
dual in a managerial or supervisory role, in an institu- 
tional healthcare organization with management and 
supervisory mandates. Both needed to be identified as 
responsible for decisions on behalf of a large jurisdiction 
or organization. Studies had to indicate decision makers' 
use of systematic reviews in either health policy or man- 
agement decisions, or on a broader range of policy or 
management decisions if these include health policy or 
health management decisions in some capacity. Studies 
of decision making in relation to an individual patient 
were excluded. Any study that examined interventions 
intended to increase seeking, appraising, or applying evi- 
dence from systematic reviews (as a source document) 
by a health policymaker or manager was included. The 
use of products or tools derived from systematic reviews 
(e.g., guidelines, evidence summaries) was not consid- 
ered, because our focus was the use of systematic 
reviews. 



Primary outcomes of interest were: the choice to 
endorse evidence-based problem formulations, pro- 
grams, and services (and drugs) to address problems; 
health system arrangements that get effective programs 
and services to those who need them; and implementa- 
tion strategies for selected policies, programs and ser- 
vices (e.g., for a tobacco cessation intervention, program, 
or policy), as well as the choice not to endorse those 
not supported by the best available evidence by a health 
policymaker or manager. Two people independently 
screened all titles and abstracts for inclusion. If at least 
one person selected the article, it was identified for full- 
text retrieval. 

Data extraction and quality assessment 

Standardized data abstraction forms were developed 
drawing on the Cochrane EPOC (Effective Practice and 
Organisation of Care Group) data abstraction form [9] 
and pilot tested by the review team using the protocol 
to guide primary and secondary outcomes. The follow- 
ing information was extracted from each article: setting, 
country, health area addressed, frequency and timing of 
the intervention, duration of the intervention, format of 
the intervention (e.g., web-based, person-to-person 
contact), known effectiveness of the intervention for 
changing behaviours (e.g., does the study use an evi- 
dence-based intervention), nature of the intervention 
(e.g., training, mode of payment, team approach), num- 
ber of components included in the intervention, source 
and authors of the intervention (e.g., professional organi- 
zation, governmental agency), mode of delivery (e.g., 
individuals or groups), reliability and validity testing of 
outcome measurement tools, and adherence (e.g., with- 
drawals, drop-outs). Two reviewers independently 
assessed each study and undertook data abstraction 
directly from primary studies. Disagreements were dis- 
cussed until consensus was achieved. A third reviewer 
was available if consensus could not be reached. 
Authors were contacted for missing data or when clarifi- 
cation was required. 

Two independent reviewers assessed the methodologi- 
cal quality of all studies that were included for data 
abstraction. Any discrepancies in ratings were resolved 
by discussion. Reviewers were not blinded to study 
author, institution, or journal, as evidence indicates that 
little benefit is achieved through blinding [10,11]. The 
criteria described in section 6.4 of the Data Collection 
Checklist from the Cochrane EPOC (available at: http:// 
www.epoc.cochrane.org) was used for randomised trials, 
and a modified Downs and Black tool [12,13] was used 
for observational studies. The criteria used to assess ran- 
domised trials were concealment of allocation, follow up 
of professionals, follow up of patients or episodes of 
care, blinded assessment of primary outcome(s), baseline 
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measurement, reliable primary outcome measure(s), and 
protection against contamination. The criteria used to 
assess observational studies were reporting, external 
validity, and internal validity. 

Results 

Initial searches of electronic databases identified 17,819 
records. After removing duplicates, 11,297 records were 
examined to determine potential relevance. Of these, 
263 were identified as related to health policymakers 
and managers, and 37 full-text articles were retrieved 
and screened. After screening all studies, 33 articles 
were excluded due to not having a relevant intervention. 
Three articles reporting on different aspects of one 
study that involved two cross-sectional surveys and one 
article describing a randomised controlled trial met the 
full inclusion criteria (Figure 1) [14-17]. 

All studies were identified as focusing on seeking, 
appraising, and applying evidence from systematic 
reviews in decision making by policy or managers 
(Table 1) [14-17]. All surveys took place in Ontario, 
Canada [14-16]. 

One randomised trial encouraged health department 
personnel in Canada to access systematic reviews on 
healthy body weight promotion in children using one of 
three potential interventions [17]. A meta-analysis of 
study outcomes was not possible due to the heterogene- 
ity in the format of the interventions, the settings, and 
healthcare areas being addressed. It is only feasible to 
provide a narrative description of the results using a 
strategy suggested by the Best Evidence Medical Educa- 
tion Collaboration [18] and based on the validity of the 
individual studies. 

Participants and settings 

Public health policymakers and managers were identified 
as the population examined in all of the studies [14-17]. 
Ciliska et al. [14] described the original research project 
conducted in 1996, where attempts were made to iden- 
tify all public health policy decision makers in Ontario, 
Canada. This was done by contacting the Public Health 
Branch and every public health department in the pro- 
vince, and asking them to identify all relevant personnel. 
270 individuals were identified and invited to take part 
in the survey. There are discrepancies in reporting, as 
277 individuals are later reported as being eligible to 
participate. Of these, 242 (87%) people completed the 
first survey and 225 (81%) completed the follow up sur- 
vey three months later. In 1998, participants were con- 
tacted again. They were invited to complete another 
survey if they had taken part in the previous study and 
were still employed in a public health department. Of 
these, 147 participants agreed to participate, and 
responses were received from 141 participants [15,16]. 



In the randomised trial, Dobbins et al. [17] invited all 
health departments in Canada to participate with fol- 
low-up data obtained from 88 out of 108 departments. 

Interventions 

Three articles report on one intervention where public 
health policymakers are offered the opportunity to 
receive five relevant systematic reviews in 1996, and fol- 
lowed up at three months [14] and two years [15,16]. 
The initial survey asked policymakers and managers if 
they would like to receive a one-time delivery of the five 
systematic reviews [14]. The systematic reviews offered 
to the participants covered the public health topics on 
the effectiveness of: home visiting; community develop- 
ment projects; maternal-child interventions; adolescent 
suicide prevention; and heart health projects [14-16]. 
Among other questions, all follow-up surveys specifically 
asked about the use of the systematic reviews to make a 
decision related to policy [14-16]. 

The randomised trial consisted of health departments 
receiving one of three interventions: access to an online 
registry of systematic reviews, tailored messages plus 
access to the online registry of systematic reviews, and 
tailored messages plus access to the registry along with 
a knowledge broker who worked one-on-one with deci- 
sion makers over a period of one year [17]. Data col- 
lected for evaluation included effects on global 
evidence-informed decision making, and effects on pub- 
lic health policies and programs. Global evidence- 
informed decision making is the extent to which 
research evidence was considered in recent program- 
planning decisions related to healthy body weight pro- 
motion. Public health policies and programs was a mea- 
sure derived as a sum of actual strategies, policies, and 
interventions for healthy body weight promotion in chil- 
dren being implemented by the health department cal- 
culated in the timeframe spanning from baseline to 
post-intervention, which was approximately 18 months. 

Effect of the intervention 

Ciliska et al. [14] report that three months after the 
intervention, 91% of participants remembered receiving 
systematic reviews. Of these, 23% said it played a part in 
program planning or decision making. Of this group, 
57% reported it influenced recommendations made to 
others, and that 64% of those recommendations were 
accepted [14]. There is no reporting of examples around 
how information from the systematic reviews was incor- 
porated into a policy or program [14]. The two articles 
by Dobbins et al. [15,16] describe the survey conducted 
two years later. Recipients of this survey indicated a 
63.1% utilization rate of at least one of the systematic 
reviews in the two years since they had been in contact. 
The significant predictors for use of systematic reviews 
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Figure 1 Flow diagram of systematic review to identify eligible studies 



are: the position of the participant - being a director 
(OR 9.82, 95% CI 1.48 to 65.32) or manager (OR 14.04, 
95% CI 2.22 to 88.96) as compared with medcial and 
associate medical officers of health; having the expecta- 
tion to use reviews in future (OR19.25, 95% CI 2.44 to 



151.99); having the perception that reviews would over- 
come limited critical appraisal skills (OR 3.36, 95% CI 
1.36 to 8.31); and that reviews were easy to use (OR 
3.01, 95% CI 0.98 to 9.29) (Dobbins 2001a). Although 
141 people agreed to participate in this survey, only 88 
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complete surveys were available for statistical analysis 
that identified these predictors of using systematic 
reviews. Similar to the reporting by Ciliska et al. [14], 
the implementation of specific policies and programs is 
reported but no specific examples are given [15,16]. 

In the randomised trial, Dobbins et al. [17] were not 
able to show a significant effect of any of the interven- 
tions on global evidence-informed decision making (p < 
0.45). With regards to effects on public health policies 
and programs, health departments that received tailored 
messages plus access to the online registry of systematic 
reviews improved significantly from baseline to follow- 
up (p < 0.01) in comparison to the groups that had 
access to the online registry only, or the groups that had 
access to the registry and also had a knowledge broker 
working with them. Research use was further examined 
by asking participants whether they were in what 
authors described as 'low' (four of seven on a seven- 
point Likert scale) versus 'high' (six of seven on a seven- 
point Likert scale) research cultures within their organi- 
zations. They observed that knowledge brokers along 
with access to systematic reviews showed a trend 
towards a positive effect when organizational research 
culture is perceived as low. However, health depart- 
ments with a low organizational research culture only 
benefited slightly when they received the tailored mes- 
sage plus access to the online registry of systematic 
reviews, yet showed great improvements when the 
research culture was high. These relationships need to 
be further explored, but they do offer support to the 
importance of organizational factors. 

Quality Assessment Results 

Quality assessments of the studies indicate that clustering 
effects and other problems that could put them at a risk 
of bias were identified as sufficient to affect interpretation 
of results. The paper by Ciliska et al. [14] includes lim- 
ited details on the study design and no details on sample 
size for the initial follow-up survey. Questions that did 
not test well during reliability testing were re-worded but 
not further tested [14]. Dobbins et al. [15,16] identify the 
large number of independent variables combined with a 
small sample size as a limitation to their second follow- 
up survey. The authors acknowledge that the large num- 
ber of independent variables may have resulted in some 
variables being significant due to chance alone. Thus, the 
predictors they describe as having a relationship with 
using systematic reviews, such as the position of the par- 
ticipant {e.g., being a manager or director), must be inter- 
preted with this caution in mind. The lack of 
independence among subjects within groups (or cluster- 
ing effects), along with results being generalizable only to 
public health professionals making decisions in Ontario, 
Canada were recognized as a limitation of the study. The 



trial by Dobbins et al. [17] describes adequate sequence 
generation and allocation concealment, and addresses 
incomplete outcome data. The authors report that there 
may have been discrepancies in the ability of the inter- 
ventions to be implemented, and the rate of successful 
intervention may have differed across the three interven- 
tion groups [17]. It is uncertain what effect this has on 
the study because the interventions were assessed 
according to group, with the effect being group-specific. 
Investigators were limited by participants' ability to self 
report in outcome measures, e.g., research use, as well as 
the use of one representative individual for each organi- 
zation to provide data. For the group that worked with a 
knowledge broker, 30% of participants had limited or no 
engagement with the knowledge broker, thus the authors 
recommend caution with the generalizability of these 
results. 

Discussion 

To our knowledge, this is the first systematic review of 
the literature on the impact of interventions for seeking, 
appraising, and applying evidence from systematic 
reviews in decision making by health policymakers or 
managers. The review of four articles revealed a paucity 
of experimental research on interventions that encou- 
rage policymakers and managers to use systematic 
reviews in decision making. Three of the articles report 
data from one intervention that distributed five systema- 
tic reviews to health policymakers and managers in pub- 
lic health, with one follow up survey conducted after 
three months, and another follow-up survey adminis- 
tered two years later. From the two follow-up surveys, 
authors were able to report that at three months, 23% 
of participants stated the reviews played a part in pro- 
gram planning or decision making [14]. However, it is 
not possible to determine if participants did use the 
results given this is based on self report. Two years 
later, 63% of respondents reported they had used at 
least one of the systematic reviews in the past 2 years to 
make a decision [15,16]. However, data on the propor- 
tion of the sample size that responded to these ques- 
tions as it relates to the original survey by Ciliska et al. 
[14] is not reported, and this lack of context may alter 
the understanding of results. For instance, one-third of 
the respondents from the Ciliska et al. [14] survey did 
not participate in this follow-up study two years later 
[15,16]. Several factors further create challenges in inter- 
preting the data presented in the three articles, includ- 
ing the lack of a control group, methodological 
limitations relating to small sample size, clustering 
effect, and limited detail in the reporting of data. The 
randomised trial suggests that tailored, targeted mes- 
sages plus online access to systematic reviews can be an 
effective strategy for evidence-informed decision making. 
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Several limitations in this review should be considered. 
The literature in this area is poorly indexed. This chal- 
lenge was acknowledged in the choice to conduct a 
scoping review as a strategy to understand the overall 
state of research activity in the area of the use of sys- 
tematic reviews in healthcare decision making. Scoping 
reviews are often undertaken when an area has little 
published research available, or the area is poorly under- 
stood [19]. The search strategy for the scoping review 
allowed for a very broad search and examination of over 
10,000 articles. The small number of studies available 
for assessment indicates the difficulty in summarizing 
and identifying key aspects in successful strategies that 
encourage health policymakers and managers to use sys- 
tematic reviews in decision making. The limited empiri- 
cal data render the strength of evidence weak in relation 
to the effectiveness and the types of interventions that 
encourage health policymakers and managers to use sys- 
tematic reviews. Second, this review is limited by the 
reports of methods from the included studies. 

Conclusions 

This review found four relevant articles which provide 
limited evidence that the interventions outlined changed 
decision making behaviour. Overall, there is insufficient 
evidence to support or refute interventions for seeking, 
appraising, and applying evidence from systematic 
reviews in decision making by health policymakers and 
managers, however the intervention describing the use 
of tailored messages is promising. Considerations for 
future research include examining the circumstances 
and contexts under which systematic reviews are most 
effective. This includes how systematic reviews are 
accessed, when they are used (e.g., different points in 
the process of developing policies), identifying the types 
of reviews needed in concert with the stage of policy- 
making (effectiveness versus process evaluation), under- 
standing more about the local applicability of systematic 
reviews, and the specific characteristics that make sys- 
tematic reviews easy to use in terms of presentation and 
format of information (e.g., grading entries, providing 
contextual information) [5,20-22]. 
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